A continuum among logarithmic, linear, and exponential functions, and its potential to improve generalization in neural networks

摘要

We present the soft exponential activation function for artificial neuralnetworks that continuously interpolates between logarithmic, linear, andexponential functions. This activation function is simple, differentiable, andparameterized so that it can be trained as the rest of the network is trained.We hypothesize that soft exponential has the potential to improve neuralnetwork learning, as it can exactly calculate many natural operations thattypical neural networks can only approximate, including addition,multiplication, inner product, distance, polynomials, and sinusoids.